We demonstrate a digital optical communication system based on minimum shift keying (MSK) signal transmission with coherent detection. 5-Gb/s MSK signal can transmit over a 160-km standard single mode fiber (SSMF) without phase compensation. At the receiver, we use data-aided channel estimation and frequency domain equalization (FDE) techniques in the digital signal processing (DSP) algorithm, then analyze its performance characteristics compared with quadrature phase shift keying (QPSK) format. The simulation results show that the MSK format will be a potential candidate for next-generation access network.